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An Initial 
Investigation 

University students (n « 69) enrolled in an introductory psychology course 
at an historically Black university coynpleted nine curriculum-based mea- 
surement probes. Results for each student were graphed and trendhnes uvre 
calculated. lYendUne direction uas examined to determine whether a possible 
formative assessment technique might be developed for use at the university 
level. Suggestions for further research concluded the investigation. 

A 

X JL sscssmcnt is an integral part of 
effective education. Without valid, reliable and efficient assessment, 
teachers have no idea whether students have learned what they have 
been taught. Research findings provide support for Curriculum-Based 
Measurement as effective and efficient classroom assessment (c.g., 
Fuchs, Fuchs, Sr Hamlctt, 1994; Gable, Arllcn, Evans, & Whinncry, 
1997). 

McDaniel, Evans-Hampton, Skinner, Hcnington, and Sims (2002) 
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described the characteristics of curriculum-based measurement (CBM) 
as procedures that arc 

...useful in making educational decisions for specific students. Spe- 
cifically, CBM procedures are brief; educators can construct multiple 
parallel forms from the students' curricula; and results yield rate data 
that are extremely sensitive to changes in academic skills ... Thus, 
CBM data can be used to evaluate students' learning rates frequently 
and over brief periods of time.... These characteristics allow educators 
to use CBM to compare more than one instructional intervention to 
determine which intervention maximizes learning rates for individu- 
als. (p. 1 ) 

A large body of research (Dcno, 1985; Fuchs, Dcno, & Mirkin, 1984; 
Fuchs & Fuchs, 1986; Fuchs & Fuchs, 1988; Kim, 1998) has provided sup- 
port for using CBM, a set of standard, simple, objcctivcly-scorcd, short- 
duration fluency measures, or ‘probes* to measure student achievement 
in important areas of basic skills or literacy, such as reading, spelling, 
written expression, and mathematics. The data from these measures 
arc designed to frequently monitor students' growth in important skills 
domain relevant to school outcomes (Shinn S' Baimonto, 1998), and, 
when graphed, provide an casy-to-intcrprct means of determining stu- 
dent progress (Council for Exceptional Children, 2005). These probes 
arc superior to teacher-made quizzes because they can be constructed 
with adequate reliability, and therefore consistently compare the same 
construct across time and instruction. Tfeachcr-madc quizzes may not 
be constructed to measure reliably, and may therefore not yield data 
that can be compared over time (Dcno ct al., 1982). 

An equally large body of research has established support for the 
technical adequacy, validity, standardization, and reliability' of CBM to 
formativcly assess student progress (Bean & Lane, 1990; Dcno ct al., 
1982; Espin & Dcno, 1993). Tliis technical adequacy is useful not only 
to researchers, but also to teachers who wish to have an indication dur- 
ing the class that their students arc on the path toward mastering the 
material being taught. 

In CBM, academic performance is sampled through the use of dynamic 
direct observation procedures. All CBM scores arc obtained by counting 
the number of correct and incorrect responses made in a fixed time 
(Dcno, 2003). Scores, then, provide a measure of the level of fluency' 
on a student's skill level. 

One of the most distinctive and important features of CBM is that 
performance is reliably sampled repeatedly across time. As a dymamic 
indicator of student learning, CBM is sensitive to differences in student 
learning. According to Shinn and Baimonto (1998), 
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Commercially available norm-referenced achievement testa are con- 
structed to be sensitive to differenced among individuals.... CBM, like 
commercially available teats, is also sensitive to differences among in- 
dividuals. However, unlike commercial norm-referenced achievement 
testa, CBM is dynamic (i.e., sensitive) to differences uithin a person 
over time. They are designed to be sensitive to the short-term effects 
(Le., 4-6 weeks) of instruction. When a student's skills change, the 
measures will detect this growth. . . . This sensitivity allows for a student S 
learning to be detected and documented, (p. 6) 

CBM is a formative evaluation tool. Instead of evaluating a student’s 
instructional program at the end of an instructional period (i.e., summa- 
tivc evaluation), when it is too late to change an on-going program, CBM 
is used to assess a student’s progress continuously during instruction 
(formative evaluation). CBM allows the teacher to make regular— and 
repeated— decisions about whether student progress is satisfactory or 
unsatisfactory. If CBM show’s that the student is progressing satisfacto- 
rily, the program is maintained. Should CBM indicate that the student 
is not making satisfactory progress, the instructional program may be 
changed in some meaningful way with the goal of improving student 
outcomes (Shinn & Baimonto, 1998). 

Research on CBM in the assessment ofbasic skills has been conducted 
for the most part on students in the elementary’ and middle school grades 
(Fuchs, Fuchs, & Maxwell, 1988). Students in both special education 
(Frank & Gcrkin, 1990) and general education (Baker S' Good, 1995) have 
participated in this research. Results from research in which CBM was 
employed suggested that because CBM uses stimulus materials drawn 
from a student's local curriculum, students from minority’ backgrounds 
may be more adequately assessed by’ CBM than by achievement or in- 
telligence tests that use a broad sample (usually national) of students 
to determine norming distribution (Shinn, Collins, S' Gallagher, 1998). 
Dcno (1985) suggested that because a problem exists when a student 
docs not perform the academic behaviors) expected in a particular 
curriculum, such problems arc seen as situational in CBM, rather than 
inherent in a particular student. 

The majority of CBM research at the secondary’ level has been con- 
ducted in the area of content-area learning, with a small body r of work 
conducted in written expression and mathematics (Shinn S' Baimonto, 
1998). Dcno (2003) reported research findings related to the use of 
measures for determining general reading proficiency’ at the secondary’ 
level, as well as research on measures focused solely’ on content-area 
reading. 

Allindcr (1995), Bean and Lane (1990), and Sticht (1999) reported 
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research conducted using CBM in adult education, but apparently no 
studies have been conducted using CBM in the university classroom. An 
extensive search of the literature, using both publicly available search 
engines (Google and Google Scholar; Search ERIC.org), and tor-pay 
Internet libraries (Highbeam Research), yielded no research articles 
on this subject. 

In line with the research available on using CBM with adult learners, a 
short review of the use of CBM with secondary (specifically high school) 
students would seem to otter at least a partial background for studying 
CBM measures for university students. This use of the CBM research 
should, howcvci; be used with caution because the two populations may 
be sufficiently different so that what is known about sccondaiy students 
may not hold true with university students. 

Espin and Tindal (1998) reported on three CBM measures (read aloud 
from text, a doze-like procedure called maze, and vocabulary matching) 
that have been developed to assess secondary-level students’ progress in 
general reading skills and content-area reading skills. All three of these 
CBM measures proved to be adequate predictors of student performance, 
but Espin and Deno (1994-1995) and Espin and Foegen (1996) found 
that, of the three, the vocabulary matching task was a better measure 
of performance on reading and content-area learning tasks than arc the 
other two. As Espin and Tindal (1998) suggested, "As students increase 
their knowledge in a content area... we could imagine that they would 
also increase their knowledge of words specific to that area” (p. 231). 
Tbc present investigation was intended to further explore the relation 
of monitoring student progress using CBM to content learning. 

The investigators of this study considered three questions: 

1. Did reading skill, as measured by the Accuplaccr (College 
Entrance Examination Board, 2002) reading test, vary with 
whether there was a positive relationship (ascending trend- 
line) or negative relationship (descending trendline) in a 
participant's graph of the results of CBM probes? 

2. Was there any relationship between the direction of the trend- 
fine of the participants graph of CBM probe results and the 
grade the participant earned in the class? 

3. Over the course of the class, was there any trend in the num- 
ber of ascending, descending, or horizontal trendlines across 
the class? 

Number 3 relates to the second part of Espin and Tindal's (1998) and 
Shinn's (1998) observation that students who experience difficulty in 
content-area reading may be placed into two groups: those with difficul- 
ties due to general reading deficits and those whose difficulties were 
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due to reading deficits specific to the content area. A preponderance of 
ascending trendlines in student graphs over the duration of the course 
might indicate that class participants were increasing their reading 
skills in areas specific to the course content. That is, in terms of the 
Espin and Tindal (1998) and Shinn (1998) observation, over the course 
of the class, ascending graph trendlines might indicate that students 
were gaining a better understanding of the course material, thereby 
improving their reading skill in this specific content area. Whether a 
student’s trendline was ascending, descending, or remaining horizontal 
might be an indication of that student's success in learning the class 
material. If validated, this tool would give an instructor an easy-to-use, 
straightforward, but reliable and valid, instrument for formatively 
ascertaining an individual student s progress throughout the class. De- 
scending or horizontal trendlines would be an •'early warning system* of 
student difficulty in the class. This 'early warning* would then prompt 
additional instructor intervention before the student had lost out on so 
much understanding of the class material that passing the class would 
become problematic. 

Method 

Participants 

Sixty-nine university undergraduate students (22 men and 47 women, 
mean age * 19.7, SD * 2.3), who were enrolled in two introductory' 
psychology courses at an historically Black college located in the Mid- 
Atlantic area of the East Coast, agreed to be participants in this project. 
Prior to collecting data, approval was obtained from the University's 
Institutional Review Board for the project, and all participants signed 
informed consent forms. 

Demographic backgrounds 

Participants self-reported their racial category as African American 
(72%), White (9%), and Other (1 %), Other plus African American (7%). 
One participant refused to classify' him/hcrsclf. The combined mean 
overall GPA for all participants was 2.7, on a grade scale of 4 (A) to 1 
(F). The participant group was composed of fifty freshmen (72%), 13 
sophomores (19%), six (9%) juniors, and one (1 %) senior. (Tbtals do not 
add up to 100% because of rounding.) 

Procedure 

The investigators used the results of a reading test required of all stu- 
dents entering the University to estimate general reading ability of the 
participants. The investigators then constructed and administered probes 



so 


Journal of College Reading and Learning, 37 ( 1 ) f Jviil 2006 


to the participants. Finally, the investigators recorded and processed 
the resulting data. Participants were not paid for their participation but 
were promised an addition of five extra points to their class grade for 
every vocabulary' probe they completed. A students overall class grade 
was calculated by determining the ratio of the total points the student 
earned on exams, papers, and other assignments, to the total number 
of possible points. The maximum of 45 points a student could cam by 
completing all nine probes was in addition to all other points possible. 
Since the total number of points offered in the class was 650, the most 
that these "extra* points could account for was 7% of a students grade, 
assuming that the student earned all other available points. 

Reading ability. As a means of estimating participants’ reading abil- 
ity, Accuplaccr (College Entrance Examination Board, 2002) reading 
scores and percentile ranks were obtained for all participants. Upon 
entrance to the university, all students (except for transfer students) 
were required to take Accuplaccr placement tests to determine whether 
or not the student needed to be enrolled in remedial classes. The Acc- 
uplaccr tests had a total possible score of 120, and had been used in a 
variety of higher-education settings throughout the country. The test 
also provided percentile ranks, which were determined to be of more 
immediate interest than the scores, because the percentiles represent 
a national sampling and, therefore, allow for comparison with other us- 
ers of the test. Thble 1 is provided to illustrate results from the reading 
subtest of the Accuplaccr for the participants in this study. 


Thble 1 

Means and Standard Deviations of Accuplaccr Reading Scores and 
Percentile Rank for 69 Students 



Beading Scores 

Reading Percentiles 

Means 

69 

39 

Standard Deviations 

18 

5 


Since the investigators desired to aggregate the scores of the two classes 
in a effort to strengthen any findings that might result from this study, 
Accuplaccr scores for the two classes were compared. The results of this 
comparison arc provided in Thble 2. 
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Thble 2 

Means and Standard Deviations ofAccuplaccr Reading Scores and 
Percentile Ranks for Class 1 (38 Students) 


Reading Scorns Reading Percentiles 


Means 67 44 

Standard Deviations 13 13 


Means and Standard Deviations ofAccuplaccr Reading Scores and 
Percentile Rajiks for Class 2 (39 Students) 


Reading Scores Reading Percentiles 


Means 72 35 

Standard Deviations 16 16 


Since the reading scores for Class 1 was within 10% of Class 2, the 
investigators felt that the two classes were sufficiently dose in measured 
reading skill to be compared. Scores for both classes were therefore 
aggregated. 

Probes. Generally following Espin and Tindal (1998, p. 226), a set of 
nine vocabulary probes was created by first scanning each page of the 
glossary of the textbook used in the psychology dass (Myers, 2004) into 
a computer file. Each glossary page was represented by a separate file. 
Next, each file was assigned a number starting with the number one. 
'Hicn each vocabulary word (along with its definition) on each page 
was assigned a sequential number, also starting with the number one. 
Finally, using a random number generator (Naacc, n.d.), three sets of 
random numbers were generated. The first set was used to determine 
the page of the textbook glossary' from which a word would be selected. 
The second set was used to determine which word and definition from 
that glossary’ page would be selected. The vocabulary' words were placed 
on the probe in the order in which thc\ r were selected. The third set 
was used to determine the place on the probe page in which a defini- 
tion would fall. Each set of three numbers was placed in a horizontal 
row on a three-column matrix. Each matrix had 21 rows, one for each 
word and its definition. 

The probes were constructed by selecting the first number in the first 
row and first column of the matrix. That number was used to identify a 
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computer file that represented a page from the textbook glossary. The 
second number in the same row was used to determine which word on 
the glossary page would be used first in the probe. Each glossary page had 
two columns. Starting at the top of the left column, words were counted 
down the page until the number represented by the second number in 
that same row was reached. Rules were developed to account for those 
occasions when the same word was selected for more than one probe. 
Finally, the third number in the same row of the matrix was used to 
determine the order of the selected word’s definition on the probe page. 
Tbc second word in the probe was placed below the first, and was cho- 
sen and placed in the same manner by selecting the first number in the 
second row of the matrix and using that to identify from which glossary 
page that second word would be chosen. The third number in the second 
row was used to determine the sequence of the second word’s definition 
on the probe. Each definition was numbered, with the top definition 
assigned the number one, the second the number two, and so on, until 
there were 20 words and 21 definitions selected and placed on the probe. 
Tbc definitions were numbered one to 21. One more definition than 
the number of vocabulary words was included to discourage identifying 
definitions by a process of elimination. The result was two columns, on 
the left the one for words and on the right the other for definitions. Tb 
the immediate left of each word was an underlined space. Tb complete 
the probe, the participant would place the number of the definition he 
or she chose on that underlined space so that the vocabulary word was 
matched with a definition. A space for the participant’s name was pro- 
vided at the top of the probe, as well as a space for the date the probe 
was completed. A cover sheet was attached as a first page for each probe 
to allow the experimenter to control when the participant first saw the 
probe. Nine probes were constructed in this manner. 

Probe administration. Probes were administered in class once per 
week, for a total of nine probes for the entire semester. At the beginning 
of the semester, the probe administrator described what was asked of 
the participants, saying that 1) the participants were asked to identify as 
many of the listed vocabulary words as possible by placing the number of 
the correct definition next to the number of the word; 2) the participants 
had five minutes in which to complete this identification; 3) participants 
were asked to do their best and not to worry if they did not complete 
the matching of all vocabulary words to the definitions; and 4) partici- 
pants were reminded that they would receive five extra points on their 
final grade for each probe they completed. As the semester continued 
and participants became more aware of the expectations, only a short 
summary of these expectations was given at each probe administration 
to remind participants. 
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Hie procedure for administering a probe was as follows: Graphs (which 
had been prepared beforehand) for each participant were distributed. 
Each participant had a personal graph so that the participant could chart 
his/her progress across probes. This participant graph was identical to 
the graph that the probe administrator kept on each participant to keep 
track of the number of vocabulary words corrccdy identified in each 
probe. Each participant graph contained the participant s name, a small 
vertical matrix to match probe scores to the date that a probe was given, 
and a graph. The vertical axis of the graph was labeled -Items Correct, * 
and the horizontal axis was labeled -Probe Number/ Probes were then 
passed out to all participants in the class, with the request from the 
probe administrator for participants not open their probes until told to 
do so. When all was ready, the probe administrator announced, -Please 
begin,* while simultaneously starting a small countdown timer set for 
five minutes. Participants worked on their probes until the countdown 
timer's alarm sounded. At this point, the probe administrator asked par- 
ticipants to stop, close their probes, and exchange completed probes with 
another participant. The probe administrator unveiled a chart showing 
which vocabulary word number corresponded to which vocabulary’ num- 
ber. Participants were asked to calculate the total number of correctly 
matched vocabulary words and definitions for the original participant's 
probe, place this number on the space on the probe cover sheet pro- 
vided, and pass the probe back to its owner. The probe’s owner then 
would note the score earned on the probe, and place that number in the 
appropriate slot on the small vertical matrix that listed the appropriate 
date for that probe. The probe owner then would place the probe score 
as a dot on his/her graph and connect that new dot to a previous dot on 
the graph. As time progressed, participants could see the progress thc\ f 
made by determining whether the direction of the graph’s line ascended 
or descended. The probe administrator then would collect the completed 
probes and participant graphs, and the class would resume. 

Data recording and processing. Tb record participants' scores, data 
were transferred from the probe sheets to a graph identical to the par- 
ticipants’, but under the control of the probe administrator. The probe 
administrator checked each participant’s probe to determine whether 
the data had been correctly collected and recorded. Data points were 
then recorded on the small vertical matrix of each participant's probe 
administrator graph in the same way that participants recorded their 
data, and the data point was transferred to the appropriate participant’s 
administrator's graph. 

At the end of the data-gathcring period, data for each participant was 
transferred from the paper graphs maintained by the probe administra- 
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tor to individual graphs that were created using Microsoft Office Excel 
2004, v. 11.2.5 spreadsheets. A regression trendline was calculated for 
each graph. Then the participant's name, Accuplacer reading score and 
percentile rank, the participant’s class (freshman, sophomore, etc.), and 
the grade the participant earned in the class were added to the graph. 

Results 

Reading scores vs. graph trendlines. Visual analysis of the participants' 
graphs was conducted by noting the direction of each participant s trend- 
line: ascending, descending, or horizontal. For the use of visual analysis 
in the interpretation of graph trendlines, see Homer ct al. (2005) and 
Parsonson and Baer (1992). 

Tb investigate the first relationship (i.c., Did reading skill, as mea- 
sured by the Accuplacer reading test, correlate with graph trendline 
direction?), a correlation between Accuplacer reading scores and graph 
trendline direction (1 « up, 2 ■ down, 3 ■ horizontal) using CORREL 
from the statistical function category’ of Microsoft Excel 2004 v. 11 .2.5 
was performed. The result was a correlation of 0.05 between Accuplacer 
reading scores and graph trendline direction. The conclusion that the 
investigators drew from this data was that there appeared to be little 
relationship between direction of trendline (either ascending or descend- 
ing) and measured reading skills for these participants. 

In addition, an analysis was made seeking to establish whether a cor- 
relation existed between Accuplacer reading scores and the students' 
graph slopes. Using the SLOPE statistical function of Microsoft Excel, 
the slope of each student’s probe number versus probe score was cal- 
culated. Using the CORREL statistical function of Microsoft Excel 2004 
v. 11.2.5, a correlation between Accuplacer reading scores and graph 
slopes was performed, resulting in a correlation of -0.13. This appeared 
to confirm the lack of relationship between Accuplacer reading scores 
and graph trendlines. 

Grades vs. trendline direction. Tb investigate the second question 
(i.c., Was graph trendline direction related to class grade?), the number 
of ascending trendlines, descending trendlines, and horizontal trendlines 
were counted for each grade category (i.c., As, Bs, Cs, Ds, Fs). Tkblc 2 
displays the results. 

These results indicate that of the 33 participants who earned As, 23 
(or 70%) had graphs with ascending trendlines, and 7 (21 %) had de- 
scending trendlines. Tlircc of 33 participants (9%) had trendlines that 
were horizontal. Of those who earned Bs, 24 (or 75%) of the participants 
had graphs with ascending trendlines and 8 (or 25%) had descending 
trendlines. No participants who earned Bs had a horizontal trendline. Of 
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those participants who earned a C in the class, two (67%) had ascending 
trendlines, and one (33%) had a descending trendline. No participant 
who earned a C had a horizontal trendline. There were no Ds earned 
in this class. The one student who failed the course had an ascending 
trendline. 

Thble 2 

Direction of Trendlines Compared to Grade Category for 69 Students 


Grade Ascending Descending Horizontal Tbtal 


A 

B 

C 

D 

F 


23 

24 
2 
0 
1 


7 

8 
1 
0 
0 


3 

0 

0 

0 

0 


33 

32 

3 

0 

1 


Mode of trendline direction over the duration of the class. The third 
question— which, if any, graph trendline mode (ascending, descending, 
or horizontal) was more prevalent— was determined by comparing the 
number of student graphs that had ascending, descending, or horizon- 
tal trendlines. Of the 69 student graphs completed over the duration of 
the course, 50 displayed ascending trendlines, 1 6 displayed descending 
trendlines, and three displayed horizontal trendlines. Thble 3 displays 
these results. 


Thble 3 

Graph trendline mode for 69 students 


Ascending 

Descending 

Horizontal 

Tbtal 

50 

16 

3 

69 


Discussion 

Classroom evaluation that is valid, reliable, and easy to conduct is criti- 
cal to determining whether students arc learning what has been taught 
(Summers, Bcrctvas, Svinicki S< Gorin, 2005). This important concept 
applies equally to students in higher education as it docs to students in 
other educational environments. 
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Tliis preliminary investigation was conducted to gather data on wheth- 
er participants' Accuplaccr reading skills test scores predicted trendlines 
for CBM probe scores, whether CBM probe trendlines predicted partici- 
pants’ class grades, and whether ascending, descending, or horizontal 
trendlines were more likely to be demonstrated in student graphs. 

In this investigation, there appeared to be little, if any, relationship 
between reading skills test scores and the direction of the probes' trend- 
lines. Probe trendlines appeared to be relatively evenly distributed across 
reading skill test results, so there did not appear to be any relationship 
between Accuplaccr reading skills test scores and students’ graph trend- 
lines. There appeared to be little correlation between Accuplaccr reading 
scores and the slope of students' graph trendlines. 

Data on the second question appears to be a bit more promising, but 
still did not, in the investigators’ opinions, rise to a level of significance. 
Those participants earning passing (A, B, C) grades in the course had 
graphs with ascending trendlines (n » 49) more often than participants 
who got both passing grades and descending trendlines (n ■ 16). Lack 
of a more normal distribution of grades ( most students received A’s or 
B's, no student received a D, and only one student failed the course) 
discouraged speculation on possible relationships between trendline 
direction and grades. In addition, the graph of the one student who 
failed the class had an ascending trendline. The only relationship that 
appeared to be of any note was that students who passed the course 
were more likely to have graphs with ascending trendlines than graphs 
with descending trendlines. 

Data on the third question relates to observations by Espin and 
Deno (1994-1995) as well as Espin and Foegein (1996) who found that 
a vocabulary matching task was a measure of learning on reading and 
content-area learning tasks. That most students' graphs (50 out of 69, or 
72%) had ascending trendlines may indicate that individuals in the class 
improved their performance as the class progressed. Such a finding, if 
supported by subsequent research, might provide the basis for deter- 
mining student progress on a formative basis. Determining formative 
progress might then afford the instructor opportunities to intervene with 
strategy training, increased support, and other forms of intervention for 
students not likely to pass the course without such assistance. 

Limitations. Several limitations exist in this study. Of primary impor- 
tance is the lack of a normal distribution of grades across the classes. 
Participants who earned an A, B, or C in the class had far more ascend- 
ing graph trendlines than descending or horizontal ones, but the lack of 
Ds and paucity of Fs leaves open the question regarding what direction 
the trendlines might take. 
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Future research. Until recently, CBM procedures were not available 
for secondary school students. A modicum of data for using CBM in voca- 
tional training classes docs exist (sec Sticht, 1 999), but none, apparently, 
dirccdy applies to university students. Formative evaluation assessing 
university students’ knowledge by graphing the outcomes of weekly 
CBM probes might help identify those students who arc not learning the 
content-area knowledge at a high enough rate to ensure a passing grade. 
Instructors might propose additional support, such as small study groups 
or individual tutoring, to those students showing downward trends in 
their CBM probes. Using CBM at the university level might also help to 
reduce the dropout rate among freshman students by providing a more 
realistic picture of where such students stand in relation to a passing 
grade, as the class progresses (see Espin & Tindal, 1998, pp. 234 ff. for 
a discussion of such interventions), espccialfy if flexible, intensive in- 
tervention were offered to potentially failing students. 

More research appears to be needed to extend to the university level 
these assessment tools that have demonstrated usefulness at the el- 
ementary and secondary grade levels. Instructors— and students— have 
as great a need for indications of formative success or failure in a uni- 
versity class as do teachers and students in clcmcntaiy, middle school, 
and high school settings. 
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